feat(api): serve PromQL label lookups from ClickHouse's Prometheus HTTP API by karl-power · Pull Request #3628 · ClickHouse/ClickStack

karl-power · 2026-11-12T10:26:26Z

Summary

Label lookups (/labels, /label/:name/values) for ClickHouse-backed PromQL sources now go through ClickHouse's prometheus_api_v1 handler, the same way /query and /query_range already do. SQL against timeSeriesTags() stays as the fallback.

The HTTP API is used only when the source resolves to clickhouse-http, the server is 26.9+ (when the label endpoints shipped), and the existing handler probe passes. Otherwise lookups use SQL, so a source querying over HTTP on 26.6–26.8 keeps label autocomplete.
Adds supportsPrometheusLabelsApi (26.9+) to common-utils. The probe cache now stores { query, labels } per connection, so no extra probe request is made.
handleLabelLookup proxies with the source's database/table, start/end/limit/match[], and the ClickHouse auth and attribution headers. The chosen path is recorded on the span as hyperdx.promql.label_backend.
On the HTTP path, limit is capped at PROMETHEUS_MAX_RESULT_ROWS (100k), including when absent or 0. The handler returns all values in one row, so max_result_rows can't bound it the way it bounds the SQL path.
The SQL path's match[] error now names the HTTP handler on 26.9+ as the alternative to storing time ranges.
Separate commit: the time ranges probe now matches ClickHouse errors by code (UNKNOWN_FUNCTION/UNKNOWN_TABLE) instead of message text. isClickHouseError/getClickHouseErrorType move from mcp/tools/query/helpers.ts to utils/clickhouseErrors.ts to avoid a circular import.

How to reproduce

Starting point: yarn dev (ClickHouse 26.10). In the ClickHouse play editor, run one statement at a time:

CREATE TABLE default.hdx5420_v8 ENGINE = TimeSeries

INSERT INTO default.hdx5420_v8 (metric_name, tags, samples) VALUES
  ('up', map('job', 'api'), [(now(), 1)]),
  ('up', map('job', 'worker'), [(now(), 1)]),
  ('http_requests_total', map('job', 'web'), [(now(), 1)])

In Team Settings → Sources, add a PromQL source on "Local ClickHouse" for default.hdx5420_v8 with the backend on Auto.

On the base branch:

Open /api/v1/prometheus/label/job/values?sourceId=<id>&match[]=up (logged in).

Run SYSTEM FLUSH LOGS, then:

SELECT event_time, interface, substring(query, 1, 90) AS q
FROM system.query_log
WHERE event_time > now() - 300 AND type = 'QueryFinish' AND query_id LIKE 'hdx-api-%'
ORDER BY event_time DESC

The lookup ran as SQL: interface = HTTP, query text contains timeSeriesTags(.

On this branch:

Same request. The response is {"status":"success","data":["api","worker"]}.
Same query-log check. The lookup ran inside the handler: interface = TCP, query text starts with SELECT arraySort(groupUniqArrayArray(, and there is no timeSeriesTags( row.
Set the source's backend to Table functions and repeat: same data, and the query log shows the timeSeriesTags( SQL again.

How to test on Vercel preview

N/A — non-UI change.

References

Linear Issue: part of HDX-5420
Related PRs: stacked on feat: Add a PromQL backend setting to ClickHouse PromQL sources #3601. Builds on feat: upgrade clickhouse versions and use promql remote endpoints #3142 (query proxy) and chore: upgrade ClickHouse to 26.10 #3624 (ClickHouse 26.10).

vercel · 2026-11-12T10:26:29Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
hyperdx-oss	Ready	Preview	Nov 12, 2026 1:59pm UTC
hyperdx-storybook	Ready	Preview	Nov 12, 2026 1:59pm UTC

changeset-bot · 2026-11-12T10:26:36Z

🦋 Changeset detected

Latest commit: d4468c3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/api	Minor
@hyperdx/common-utils	Minor
@hyperdx/app	Minor
@hyperdx/otel-collector	Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

github-actions · 2026-11-12T10:28:37Z

Duplicate code analysis

No new duplicate code introduced by this PR.

jscpd compared this branch against origin/main
across app, api and common-utils. Only new clones are listed; the codebase
currently holds 108 in total (1.18%).
Run yarn dupes locally to reproduce, or yarn dupes:all for the full picture.

github-actions · 2026-11-12T10:28:40Z

🟡 Tier 3 — Standard

Introduces new logic, modifies core functionality, or touches areas with non-trivial risk.

Why this tier:

Diff size: 309 production lines changed (Tier 2 max: < 250)
Cross-layer change: touches backend (packages/api) + shared utils (packages/common-utils)
Touches API routes or data models — hidden complexity risk

Review process: Full human review — logic, architecture, edge cases.
SLA: First-pass feedback within 1 business day.

Stats

Production files changed: 6
Production lines changed: 309 (+ 367 in test files, excluded from tier calculation)
Branch: claude/hdx-5420-promql-labels-http-api
Author: karl-power

To override this classification, remove the review/tier-3 label and apply a different review/tier-* label. Manual overrides are preserved on subsequent pushes.

github-actions · 2026-11-12T10:30:24Z

PR Review

If you are a coding agent acting for the author, read this first.
Nothing below blocks merge automatically; a maintainer decides. What they will expect:

🔴 critical or 🟠 major in code this PR adds or changes: fix before asking for review.

🔵 minor in code this PR changes: your call. Fix if small, otherwise reply.

Anything about surrounding code, or asking you to widen the change (hoist a helper,
dedupe with another file, fix other call sites): never fix here, whatever the
severity. Reply with one sentence; if it is critical, say so plainly so a human sees it.

One commit per review round. After two rounds, stop and ask a maintainer to review
scope rather than addressing more automated findings. Full rule: AGENTS.md.

1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor

None could be anchored to changed lines; they are listed below.

Findings outside the changed lines

1 minor

🔵 packages/api/src/routers/api/__tests__/prometheus.int.test.ts:835 — Live match[] test can't tell whether the handler served the narrowed request → mockFetch is only reset in the outer afterEach, so by the time of narrowed the call list already holds the unfiltered /label/job/values handler call. That makes toContain pass even if the match[] request fell back to SQL. The fixture table stores time bounds, so SQL would also return ['web'] and []. Fix: call mockFetch.mockClear() at the start of lookup (or assert on toEqual for that call only), and assert handlerPaths for the out-of-window lookup too.

_{Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides.}

github-actions · 2026-11-12T10:34:01Z

Deep Review

✅ No critical issues found. The diff is a focused, defensively-written backend change: label lookups route to ClickHouse's prometheus_api_v1 handler on 26.9+ and fall back to SQL otherwise, reusing the shared probe cache and the existing hardened proxy path. Independent verification confirmed the renamed/moved symbols have no dangling references (connectionSupportsPrometheusHttpApi, proxyPath, and the relocated getClickHouseErrorType/isClickHouseError are all updated at every call site), the limit cap holds for the 0/absent/over-cap cases, and request-supplied database/table keys are stripped by the zod schema before the source's own values are applied.

🟡 P2 — recommended

packages/api/src/routers/api/prometheus.ts:910 — the catch-all handler returns raw e.message to the client, so a ClickHouse SQL error on the fallback path can surface table/schema internals in the response body; this is pre-existing but now also governs the new backend split.
- Fix: return a generic bad_data message to the client and log the underlying error, matching how the proxy path already redacts upstream detail.
- _{security, correctness}

🔵 P3 nitpicks (1)

packages/api/src/controllers/timeseriesEngine.ts:579 — isNoTimeRangesTableError now treats UNKNOWN_TABLE as expected, which also matches a genuinely missing TimeSeries table; the code comment argues the earlier Tags probe already surfaces that case, so the masking is intentional but relies on an invariant in a separate function.
- Fix: add a focused test asserting a missing-table scenario is still observable upstream, so the documented invariant is guarded against regression.

Reviewers (13): correctness, security, adversarial, reliability, api-contract, performance, kieran-typescript, testing, maintainability, project-standards, previous-comments, agent-native, learnings-researcher.

Testing gaps:

No route-level test asserts a table-function source reads labels via SQL on a 26.10 server (covered at the unit level in resolveClickhouseLabelBackend, and the HTTP cap/version-gate paths are well covered).
No assertion that arbitrary extra query keys (e.g. a ClickHouse setting name) are stripped on the label path — safe today because the zod schema strips unknown keys by default, but untested against regression.

_{No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond}

github-actions · 2026-11-12T10:34:16Z

E2E Test Results

✅ All tests passed • 465 passed • 1 skipped • 1247s

Status	Count
✅ Passed	465
❌ Failed	0
⚠️ Flaky	1
⏭️ Skipped	1

Tests ran across 4 shards in parallel.

View full report →

…TP API

The merge-base changed after approval.

karl-power added this pull request to stack #3629 November 12, 2026 10:27

karl-power marked this pull request as ready for review November 12, 2026 10:28

karl-power requested a review from a team as a code owner November 12, 2026 10:28

github-actions Bot added the review/tier-3 Standard — full human review required label Nov 12, 2026

vercel Bot deployed to Preview – hyperdx-storybook November 12, 2026 10:28

View deployment

github-actions Bot reviewed Nov 12, 2026

View reviewed changes

Comment thread packages/api/src/routers/api/prometheus.ts Resolved

Comment thread packages/api/src/controllers/promql.ts Resolved

vercel Bot deployed to Preview – hyperdx-oss November 12, 2026 10:30

View deployment

karl-power mentioned this pull request Nov 12, 2026

feat(mcp): read PromQL label catalog from ClickHouse's Prometheus HTTP API #3630

Merged

vercel Bot deployed to Preview – hyperdx-storybook November 12, 2026 11:34

View deployment

vercel Bot deployed to Preview – hyperdx-oss November 12, 2026 11:34

View deployment

pulpdrew previously approved these changes Nov 12, 2026

View reviewed changes

Base automatically changed from claude/hdx-5419-promql-source-backend to main November 12, 2026 13:56

karl-power added 3 commits November 12, 2026 14:56

refactor(api): match the time ranges probe's ClickHouse errors by code

f4bbfdb

feat(api): serve PromQL label lookups from ClickHouse's Prometheus HT…

4df8f14

…TP API

fix(api): cap label values returned by ClickHouse's Prometheus handler

d4468c3

karl-power force-pushed the claude/hdx-5420-promql-labels-http-api branch from 98823ff to d4468c3 Compare November 12, 2026 13:56

vercel Bot deployed to Preview – hyperdx-storybook November 12, 2026 13:59

View deployment

vercel Bot deployed to Preview – hyperdx-oss November 12, 2026 13:59

View deployment

pulpdrew approved these changes Nov 12, 2026

View reviewed changes

karl-power merged commit 2d17de5 into main Nov 12, 2026
29 checks passed

Navigation Menu

Repository navigation

feat(api): serve PromQL label lookups from ClickHouse's Prometheus HTTP API - #3628

feat(api): serve PromQL label lookups from ClickHouse's Prometheus HTTP API#3628
karl-power merged 3 commits into
mainfrom
claude/hdx-5420-promql-labels-http-api

karl-power commented Nov 12, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Nov 12, 2026 •

edited

Loading

Uh oh!

changeset-bot Bot commented Nov 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 12, 2026

Uh oh!

github-actions Bot commented Nov 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 12, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Nov 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

karl-power commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How to reproduce

How to test on Vercel preview

References

Uh oh!

vercel Bot commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

changeset-bot Bot commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🦋 Changeset detected

Uh oh!

github-actions Bot commented Nov 12, 2026

Duplicate code analysis

Uh oh!

github-actions Bot commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟡 Tier 3 — Standard

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Review

Findings outside the changed lines

Uh oh!

github-actions Bot commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deep Review

🟡 P2 — recommended

Uh oh!

github-actions Bot commented Nov 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

E2E Test Results

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

karl-power commented Nov 12, 2026 •

edited

Loading

vercel Bot commented Nov 12, 2026 •

edited

Loading

changeset-bot Bot commented Nov 12, 2026 •

edited

Loading

github-actions Bot commented Nov 12, 2026 •

edited

Loading

github-actions Bot commented Nov 12, 2026 •

edited

Loading

github-actions Bot commented Nov 12, 2026 •

edited

Loading

github-actions Bot commented Nov 12, 2026 •

edited

Loading